.ds ]W %
.ds ]L
.nh
.TH lsf.cluster 5 "openlava Version 2.0 - Jan 2012"
.br
.SH NAME
\fBlsf.cluster\fR
.SS \fB\fROverview
.BR
.PP
.PP
This is the cluster configuration file. There is one for each cluster, called 
lsf.cluster.\fIcluster_name\fR. The \fIcluster_name\fR suffix is the cluster 
name defined in the Cluster section of lsf.shared.
.SS Contents
.BR
.PP
.RS
.HP 2
\(bu Parameters Section
.HP 2
\(bu ClusterAdmins Section
.HP 2
\(bu Host Section
.HP 2
\(bu ResourceMap Section
.HP 2
\(bu RemoteClusters Section
.RE
.SS 
.BR
.PP
.SH Parameters Section
.BR
.PP
.PP
(Optional) This section contains miscellaneous parameters for the LIM.
.SH ADJUST_DURATION
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBADJUST_DURATION =\fR \fIinteger\fR
.SS Description
.BR
.PP
.PP
Integer reflecting a multiple of EXINTERVAL that controls the time 
period during which load adjustment is in effect.
.PP
The \fBlsplace\fR(\fB1\fR) and \fBlsloadadj\fR(\fB1\fR) commands artificially raise the 
load on a selected host. This increase in load decays linearly to 0 over 
time.
.SS Default
.BR
.PP
.PP
3
.SH ELIM_POLL_INTERVAL
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBELIM_POLL_INTERVAL =\fR \fItime_in_seconds\fR
.SS Description
.BR
.PP
.PP
Time interval, in seconds, in which the LIM daemon samples load 
information
.PP
This parameter only needs to be set if an ELIM is being used to report 
information more frequently than every 5 seconds.
.SS Default
.BR
.PP
.PP
5 seconds
.SH ELIMARGS
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBELIMARGS =\fR \fIcmd_line_args\fR
.SS Description
.BR
.PP
.PP
Specifies any necessary command-line arguments for the external LIM 
on startup
.PP
This parameter is ignored if no external load indices are configured.
.SS Default
.BR
.PP
.PP
None
.SH EXINTERVAL
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBEXINTERVAL =\fR \fItime_in_seconds\fR
.SS Description
.BR
.PP
.PP
Time interval, in seconds, at which the LIM daemons exchange load 
information
.PP
On extremely busy hosts or networks, or in clusters with a large 
number of hosts, load may interfere with the periodic communication 
between LIM daemons. Setting EXINTERVAL to a longer interval can 
reduce network load and slightly improve reliability, at the cost of 
slower reaction to dynamic load changes.
.SS Default
.BR
.PP
.PP
15 seconds
.SH HOST_INACTIVITY_LIMIT
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBHOST_INACTIVITY_LIMIT =\fR \fIinteger\fR
.SS Description
.BR
.PP
.PP
Integer reflecting a multiple of EXINTERVAL that controls the maximum 
time a slave LIM will take to send its load information to the master LIM 
as well as the frequency at which the master LIM will send a heartbeat 
message to its slaves.
.PP
A slave LIM can send its load information any time from EXINTERVAL 
to (HOST_INACTIVITY_LIMIT-2)*EXINTERVAL seconds. A master LIM 
will send a master announce to each host at least every 
EXINTERVAL*HOST_INACTIVITY_LIMIT seconds.
.SS Default
.BR
.PP
.PP
5
.SH LSF_ELIM_BLOCKTIME
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBLSF_ELIM_BLOCKTIME=\fR\fIseconds\fR
.SS Description
.BR
.PP
.PP
UNIX only.
.PP
Maximum amount of time LIM waits for a load update string from the 
ELIM if it is not immediately available. 
.PP
Use this parameter to add fault-tolerance to LIM when using ELIMs. If 
there is an error in the ELIM or some situation arises that the ELIM 
cannot send the entire load update string to the LIM, LIM will not wait 
indefinitely for load information from ELIM. After the time period 
specified by LSF_ELIM_BLOCKTIME, the LIM writes the last string sent 
by ELIM in its log file (lim.log.\fIhost_name\fR) and restarts the ELIM.
.PP
For example, if LIM is expecting 3 name-value-pairs, such as:

.PP
3 tmp2 49.5 nio 367.0 licenses 3


.PP
If after the time period specified by LSF_ELIM_BLOCKTIME LIM has 
only received the following:
.PP
3 tmp2 47.5
.PP
LIM writes whatever was received last (3 tmp2 47.5) in the log file and 
restarts the ELIM.
.SS Valid Values
.BR
.PP
.PP
Non-negative integers
.PP
A value of 0 indicates that LIM will not wait at all to receive information 
from ELIM--it expects to receive the entire load string at once. 
.PP
So, if for example, your ELIM writes value-pairs with 1 second intervals 
between them, and you collect 12 load indices, you need to allow at 
least 12 seconds for the ELIM to complete writing an entire load string. 
So you would define LSF_ELIM_BLOCKTIME to 15 or 20 seconds for 
example.
.SS Default
.BR
.PP
.PP
Undefined--LIM waits indefinitely to receive load information from 
ELIM
.SH LSF_ELIM_DEBUG
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBLSF_ELIM_DEBUG=y\fR
.SS Description
.BR
.PP
.PP
UNIX only.
.PP
This parameter is useful to view which load information an ELIM is 
collecting and to add fault-tolerance to LIM. 
.PP
When this parameter is set to y:
.RS
.HP 2
\(bu All load information received by LIM from the ELIM is logged in the 
LIM log file (lim.log.\fIhost_name\fR). 
.HP 2
\(bu If LSF_ELIM_BLOCKTIME is undefined, whenever there is an error 
in the ELIM or some situation arises that the ELIM cannot send the 
entire load update string to the LIM, LIM does not wait indefinitely 
for load information from ELIM. After 2 seconds, the LIM restarts 
the ELIM. 
.RE

.IP
For example, LIM is expecting 3 name-value-pairs, such as:


.IP
3 tmp2 47.5 nio 344.0 licenses 5

.RE
.IP
However, LIM only receives the following from ELIM:


.IP
3 tmp2 47.5

.RE
.IP
LIM waits 2 seconds after the last value is received and if no more 
information is received, LIM restarts the ELIM.

.IP
If LSF_ELIM_BLOCKTIME is defined, the LIM waits for the specified 
amount of time before restarting the ELIM instead of the 2 seconds.


.SS Default
.BR
.PP
.PP
Undefined--if LSF_ELIM_DEBUG is undefined, load information sent 
from ELIM to LIM is not logged. In addition, if LSF_ELIM_BLOCKTIME 
is undefined, LIM waits indefinitely to receive load information from 
ELIM.
.SS See Also
.BR
.PP
.PP
LSF_ELIM_BLOCKTIME to configure how long LIM waits before 
restarting the ELIM, use the parameter LSF_ELIM_BLOCKTIME.
.PP
LSF_ELIM_RESTARTS to limit how many times the ELIM can be 
restarted
.SH LSF_ELIM_RESTARTS
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBLSF_ELIM_RESTARTS=\fR\fIinteger\fR
.SS Description
.BR
.PP
.PP
UNIX only.
.PP
LSF_ELIM_BLOCKTIME or LSF_ELIM_DEBUG must be defined in 
conjunction with LSF_ELIM_RESTARTS.
.PP
Defines the maximum number of times an ELIM can be restarted.
.PP
When this parameter is defined:
.RS
.HP 2
\(bu If LIM attempts to retrieve load information from the ELIM and 
there is an error such as an invalid value for example, LIM restarts 
the ELIM. 
.RE
.PP
If the error is consistent and LIM keeps restarting the ELIM, 
LSF_ELIM_RESTARTS limits how many times the ELIM can be restarted 
to prevent an ongoing loop.
.SS Valid Values
.BR
.PP
.PP
Non-negative integers
.SS Default 
.BR
.PP
.PP
Undefined; the number of ELIM restarts is unlimited
.SS See Also
.BR
.PP
.PP
LSF_ELIM_BLOCKTIME, LSF_ELIM_DEBUG
.SH MASTER_INACTIVITY_LIMIT
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBMASTER_INACTIVITY_LIMIT =\fR \fIinteger\fR
.SS Description
.BR
.PP
.PP
An integer reflecting a multiple of EXINTERVAL. A slave will attempt to 
become master if it does not hear from the previous master after 
(HOST_INACTIVITY_LIMIT 
+\fIhost_number\fR*MASTER_INACTIVITY_LIMIT)*EXINTERVAL seconds, 
where \fIhost_number\fR is the position of the host in 
lsf.cluster.\fIcluster_name\fR.
.PP
The master host is \fIhost_number\fR 0.
.SS Default
.BR
.PP
.PP
2
.SH PROBE_TIMEOUT
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBPROBE_TIMEOUT =\fR \fItime_in_seconds\fR
.SS Description
.BR
.PP
.PP
Specifies the timeout in seconds to be used for the connect(2) system 
call
.PP
Before taking over as the master, a slave LIM will try to connect to the 
last known master via TCP.
.SS Default
.BR
.PP
.PP
2 seconds
.SH RETRY_LIMIT
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBRETRY_LIMIT =\fR \fIinteger\fR
.SS Description
.BR
.PP
.PP
Integer reflecting a multiple of EXINTERVAL that controls the number 
of retries a master or slave LIM makes before assuming that the slave 
or master is unavailable.
.PP
If the master does not hear from a slave for HOST_INACTIVITY_LIMIT 
exchange intervals, it will actively poll the slave for RETRY_LIMIT 
exchange intervals before it will declare the slave as unavailable. If a 
slave does not hear from the master for HOST_INACTIVITY_LIMIT 
exchange intervals, it will actively poll the master for RETRY_LIMIT 
intervals before assuming that the master is down.
.SS Default
.BR
.PP
.PP
2
.SH ClusterAdmins Section
.BR
.PP
.PP
(Optional) The ClusterAdmins section defines the openlava administrators 
for the cluster. The only keyword is ADMINISTRATORS.
.PP
If the ClusterAdmins section is not present, the default openlava 
administrator is root. Using root as the primary openlava administrator is 
not recommended.
.SH ADMINISTRATORS
.BR
.PP
.SS Syntax
.BR
.PP
.PP
\fBADMINISTRATORS =\fR \fIadministrator_name ...\fR
.SS Description
.BR
.PP
.PP
Specify Linux user and user group names.
.PP
The first administrator of the expanded list is considered the primary 
openlava administrator. The primary administrator is the owner of the openlava 
configuration files, as well as the working files under 
LSB_SHAREDIR/\fIcluster_name\fR. If the primary administrator is 
changed, make sure the owner of the configuration files and the files 
under LSB_SHAREDIR/\fIcluster_name\fR are changed as well. 
.PP
Administrators other than the primary openlava administrator have the same 
privileges as the primary openlava administrator except that they do not have 
permission to change openlava configuration files. They can perform 
clusterwide operations on jobs, queues, or hosts in the system.
.PP
For flexibility, each cluster may have its own openlava administrators, 
identified by a user name, although the same administrators can be 
responsible for several clusters.
.PP
Use the \fB-l\fR option of the \fBlsclusters\fR(\fB1\fR) command to display all of the 
administrators within a cluster.
.SS Compatibility
.BR
.PP
.PP
For backwards compatibility, ClusterManager and Manager are 
synonyms for ClusterAdmins and ADMINISTRATORS respectively. It 
is possible to have both sections present in the same 
lsf.cluster.\fIcluster_name\fR file to allow daemons from different 
openlava versions to share the same file.
.SS Default
.BR
.PP
.PP
lsfadmin
.SS Example
.BR
.PP
.PP
The following gives an example of a cluster with three openlava 
administrators. The user listed first, user2, is the primary administrator. 
The user group lsfgrp and the user user7 are secondary 
administrators.

.PP
Begin ClusterAdmins
.br
ADMINISTRATORS = user2 lsfgrp user7
.br
End ClusterAdmins


.SH Host Section
.BR
.PP
.PP
The Host section is the last section in lsf.cluster.\fIcluster_name\fR and 
is the only required section. It lists all the hosts in the cluster and gives 
configuration information for each host.
.PP
The order in which the hosts are listed in this section is important. The 
LIM on the first host listed becomes the master LIM if this host is up; 
otherwise, that on the second becomes the master if its host is up, and 
so on.
.PP
Since the master LIM makes all placement decisions for the cluster, it 
should be on a fast machine. Also, to avoid the delays involved in 
switching masters if the first machine goes down, the master should be 
on a reliable machine. It is desirable to arrange the list such that the 
first few hosts in the list are always in the same subnet. This avoids a 
situation where the second host takes over as master when there are 
communication problems between subnets.
.PP
Configuration information is of two types:
.RS
.HP 2
\(bu Some fields in a host entry simply describe the machine and its 
configuration.
.HP 2
\(bu Other fields set thresholds for various resources.
.RE
.SH Descriptive Fields
.BR
.PP
.PP
The following fields are required in the Host section:
.RS
.HP 2
\(bu HOSTNAME
.HP 2
\(bu RESOURCES
.HP 2
\(bu type
.HP 2
\(bu server
.RE
.PP
The following fields are optional:
.RS
.HP 2
\(bu model
.HP 2
\(bu nd
.HP 2
\(bu RUNWINDOW
.HP 2
\(bu REXPRI
.RE
.SH HOSTNAME
.BR
.PP
.SS Description
.BR
.PP
.PP
Official name of the host as returned by \fBhostname\fR(1)
.PP
The name must be listed in lsf.shared as belonging to this cluster.
.SH model
.BR
.PP
.SS Description
.BR
.PP
.PP
Host model
.PP
The name must be defined in the HostModel secton of lsf.shared. 
This determines the CPU speed scaling factor applied in load and 
placement calculations.
.PP
If you leave the model or type column blank or enter the ! keyword, 
you are indicating that the host model or type is to be automatically 
detected by the LIM running on the host.
.SH nd
.BR
.PP
.SS Description
.BR
.PP
.PP
Number of local disks
.PP
This corresponds to the ndisks static resource. On most host types, openlava 
automatically determines the number of disks, and the nd parameter is 
ignored.
.PP
nd should only count local disks with file systems on them. Do not 
count either disks used only for swapping or disks mounted with NFS.
.SS Default
.BR
.PP
.PP
The number of disks determined by the LIM, or 1 if the LIM cannot 
determine this
.SH RESOURCES
.BR
.PP
.SS Description
.BR
.PP
.PP
The static Boolean resources available on this host
.PP
The resource names are strings defined in the Resource section of 
lsf.shared. You may list any number of resources, enclosed in 
parentheses and separated by blanks or tabs: for example:

.PP
(fs frame hpux)


.PP
Optionally, you can specify a dedicated resource by prefixing the 
resource with an exclamation mark (!). A host with dedicated resources 
is not selected by LIM for a job unless a dedicated resource name is 
explicitly specified in the resource requirements for the job.
.SH REXPRI
.BR
.PP
.SS Description
.BR
.PP
.PP
(UNIX ONLY) Default execution priority for interactive remote jobs run 
under the RES
.PP
The range is from -20 to 20. REXPRI corresponds to the BSD-style nice 
value used for remote jobs. For hosts with System V-style nice values 
with the range 0 - 39, a REXPRI of -20 corresponds to a nice value of 
0, and +20 corresponds to 39. Higher values of REXPRI correspond to 
lower execution priority; -20 gives the highest priority, 0 is the default 
priority for login sessions, and +20 is the lowest priority.
.SS Default
.BR
.PP
.PP
0
.SH RUNWINDOW
.BR
.PP
.SS Description
.BR
.PP
.PP
Dispatch window during this host is accepts remote interactive tasks
.PP
When the host is not available for remote execution, the host status is 
lockW (locked by run window). LIM does not schedule interactive tasks 
on hosts locked by dispatch windows. Note that run windows only 
apply to interactive tasks placed by LIM. openlava Batch uses its own 
(optional) host dispatch windows to control batch job processing on 
batch server hosts.
.SS Format
.BR
.PP
.PP
A dispatch window consists of one or more time windows in the format 
\fIbegin_time\fR-\fIend_time\fR. No blanks can separate \fIbegin_time\fR and 
\fIend_time\fR. Time is specified in the form [\fIday\fR:]\fIhour\fR[:\fIminute\fR]. If only one 
field is specified, openlava assumes it is an \fIhour\fR. Two fields are assumed to 
be \fIhour\fR:\fIminute\fR. Use blanks to separate time windows.
.SS Default
.BR
.PP
.PP
Always accept remote jobs
.SH server
.BR
.PP
.SS Description
.BR
.PP
.PP
Indicates whether the host can receive jobs from other hosts
.PP
Specify 1 if the host can receive jobs from other hosts; specify 0 
otherwise. If server is set to 0, the host is an openlava client. Client hosts do 
not run the openlava daemons. Client hosts can submit interactive and batch 
jobs to an openlava cluster, but they cannot execute jobs sent from other 
hosts.
.SS Default
.BR
.PP
.PP
1
.SH type
.BR
.PP
.SS Description
.BR
.PP
.PP
Host type as defined in the HostType section of lsf.shared
.PP
The strings used for host types are determined by the system 
administrator: for example, SUNSOL, DEC, or HPPA. The host type is 
used to identify binary-compatible hosts.
.PP
The host type is used as the default resource requirement. That is, if no 
resource requirement is specified in a placement request, the task is 
run on a host of the same type as the sending host.
.PP
Often one host type can be used for many machine models. For 
example, the host type name SUNSOL6 might be used for any 
computer with a SPARC processor running SunOS 6. This would 
include many Sun models and quite a few from other vendors as well.
.PP
If you leave the model or type column blank or enter the ! keyword, 
you are indicating that the host model or type is to be automatically 
detected by the LIM running on the host.
.SH Threshold Fields
.BR
.PP
.PP
The LIM uses these thresholds in determining whether to place remote 
jobs on a host. If one or more openlava load indices exceeds the 
corresponding threshold (too many users, not enough swap space, 
etc.), then the host is regarded as busy, and LIM will not recommend 
jobs to that host.
.PP
The CPU run queue length threshold values (r15s, r1m, and r15m) are 
taken as effective queue lengths as reported by \fBlsload -E\fR.
.PP
All of these fields are optional; you only need to configure thresholds 
for load indices that you wish to use for determining whether hosts are 
busy. Fields that are not configured are not considered when 
determining host status. The keywords for the threshold fields are not 
case sensitive.
.PP
Thresholds can be set for any of the following:
.RS
.HP 2
\(bu The built-in openlava load indexes (r15s, r1m, r15m, ut, pg, it, io, ls, swp, 
mem, tmp)
.HP 2
\(bu External load indexes defined in the Resource section of 
lsf.shared
.RE
.SH Example of a Host Section
.BR
.PP
.PP
This example Host section contains descriptive and threshold 
information for two hosts:

.PP
Begin Host
.br
HOSTNAME   model    type   server r1m pg tmp RESOURCES     RUNWINDOW
.br
hostA      SparcIPC Sparc  1      3.5 15   0 (sunos frame  ()
.br
hostD      Sparc10  Sparc  1      3.5 15   0 (sunos)       (5:18:30-1:8:30)
.br
End Host


.SH ResourceMap Section
.BR
.PP
.PP
The ResourceMap section defines shared resources in your cluster. This 
section specifies the mapping between shared resources and their 
sharing hosts. When you define resources in the Resources section of 
lsf.shared, there is no distinction between a shared and non-shared 
resource. By default, all resources are not shared and are local to each 
host. By defining the ResourceMap section, you can define resources 
that are shared by all hosts in the cluster or define resources that are 
shared by only some of the hosts in the cluster.
.PP
This section must appear after the Host section of 
lsf.cluster.\fIcluster_name\fR, because it has a dependency on host 
names defined in the Host section. The following parameters must be 
defined in the ResourceMap section:
.SH ResourceMap Section Structure 
.BR
.PP
.PP
The first line consists of the keywords RESOURCENAME and 
LOCATION. Subsequent lines describe the hosts that are associated 
with each configured resource.
.SH LOCATION
.BR
.PP
.SS Description
.BR
.PP
.PP
Defines the hosts that share the resource
.PP
For a static resource, you must define a value here as well. Do not 
define a value for a dynamic resource.
.PP
\fIinstance\fR is a list of host names that share an instance of the resource. 
The reserved words all, others, and default can be specified for the 
instance:
.RS
.HP 2
\(bu all--Indicates that there is only one instance of the resource in the 
whole cluster and that this resource is shared by all of the hosts
.RE

.IP
Use the not operator (~) to exclude hosts from the all 
specification. For example:


.IP
(2@[all ~host3 ~host4])

.RE
.IP
means that 2 units of the resource are shared by all server hosts in 
the cluster made up of host1 host2 ... host\fIn\fR, except for host3 and 
host4. This is useful if you have a large cluster but only want to 
exclude a few hosts.

.IP
The parentheses are required in the specification. The not operator 
can only be used with the all keyword. It is not valid with the 
keywords others and default.


.RS
.HP 2
\(bu others--Indicates that the rest of the server hosts not explicitly 
listed in the LOCATION field comprise one instance of the resource
.RE

.IP
For example:


.IP
2@[host1] 4@[others] 

.RE
.IP
indicates that there are 2 units of the resource on apple and 4 units 
of the resource shared by all other hosts.


.RS
.HP 2
\(bu default--Indicates an instance of a resource on each host in the 
cluster
.RE

.IP
This specifies a special case where the resource is in effect not 
shared and is local to every host. default means at each host. 
Normally, you should not need to use default, because by default 
all resources are local to each host. You might want to use 
ResourceMap for a non-shared static resource if you need to specify 
different values for the resource on different hosts.


.SH RESOURCENAME
.BR
.PP
.SS Description
.BR
.PP
.PP
Name of the resource
.PP
This resource name must be defined in the Resource section of 
lsf.shared. You must specify at least a name and description for the 
resource, using the keywords RESOURCENAME and DESCRIPTION.
.RS
.HP 2
\(bu A resource name cannot begin with a number. 
.HP 2
\(bu A resource name cannot contain any of the following characters: 

.IP
:  .  (  )  [  +  - *  /  !  &  | <  >  @  =

.RE
.HP 2
\(bu A resource name cannot be any of the following reserved names:

.IP
cpu cpuf io logins ls idle maxmem maxswp maxtmp type model 
status it mem ncpus ndisks pg r15m r15s r1m swap swp tmp ut

.RE
.HP 2
\(bu Resource names are case sensitive
.HP 2
\(bu Resource names can be up to 29 characters in length
.RE
.SH Example of a ResourceMap Section
.BR
.PP

.PP
Begin ResourceMap
.br
RESOURCENAME   LOCATION
.br
verilog        [5@all]
.br
local          ([host1 host2] [others])
.br
End ResourceMap


.PP
The resource verilog must already be defined in the RESOURCE 
section of the lsf.shared file. It is a static numeric resource shared 
by all hosts. The value for verilog is 5. The resource local is a 
numeric shared resource that contains two instances in the cluster. The 
first instance is shared by two machines, host1 and host2. The second 
instance is shared by all other hosts.
.PP
Resources defined in the ResourceMap section can be viewed by using 
the \fB-s\fR option of the \fBlshosts\fR (for static resource) and \fBlsload\fR (for 
dynamic resource) commands.
